DOCUMENT RESUME 



ED 251 722 CG 017 850 

AUTHOR Schmidt, Frank L. 

TITLE Meta-Analysis: Implications for Cumulative Knowledge 

in the Behavioral and Social Sciences. 
PUB DATE 25 Aug 84 

NOTE 14p.; An invited address at the Ann\ il Convention of 

the American Psychological Association (92nd, 
Toronto, Ontario, Canada, August 24-28, 1984). 

PUB TYPE Viewpoints (120) — Speeches/Conference Papers (150) 

EDRS PRICE MFOl/PCOl Plus Postage. 

DESCRIPTORS *Meta Analysis; ^Psychology; P'^search Methodology; 

Research Problems; ^Social Science Research 

ABSTRACT 

The most important problem in psychology and the 
social sciences today is the failure to produce cumulative knowledge. 
This situation has led many to conclude that cumulative knowledge and 
general principles and theories may be impossible to establish in 
psychology, a conclusion suggesting that psychology can never be a 
science — only at best a technology producing answers limited in their 
applicability to specific situations. This notion has contributed to 
reductions in funding for research. Psychological researchers have 
relied on a general two-step procedure to produce cumulative 
knowledge. First, individual scientists conduct numerous studies; 
second these studies are integrated subjectively and 
non-quantitatively , published, and reviewed. However, the information 
processing task in reviewing a body of studies so that general 
principles can be drawn from them may be too complex for the unaided 
human mind. The most important recent development, therefore, is the 
development of quantitative methods which lift the information 
processing burden from the reviewer by quantitatively integrating 
findings across studies, while simultaneously correcting for the 
effects of statistical and measurement artifacts which distort study 
findings. Meta-analysis, the term for this process, corrects for the 
distorting effects of artifacts such as sampling error, measurement 
unreliability, and range restrictions. The response to these new 
methods by researchers has been gratifying, as evidenced by 
meta-analytic studies in journals, books and convention 
presentations. (JAC) 
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The most important problem in psychology and the social sciences today 
is the failure to produce cumulative knowledge. In one research area after 
another, psychologists have conducted numerous studies, and in almost every 
case it has been found that different studies give different results. Some 
studies report significant findings, some do not. Some studies support the 
hypothesis, some do not. This situation has led many to conclude that cumu- 
lative knoweldge and general principles and theories may be impossible to 
establish in psychology and the social sciences. 

If this conclusion is true, then psychology can never be a science— only 
at best a technology producing answers limited in their applicability to 
specific situations. It follows then that psychological research can never 
provide solutions to broad social problems business, education, and society- 
as had been expected by research funding organizations such as foundations and 
government agencies. The resulting frustration is expressed in a speech given 
in 1970 by Walter Mondale: 

What I have no£ learned is what we should do about these problems. 
I had hoped to find research to support or to conclusively oppose 
my belief that quality integrated education is the most promising 
approach. But I have found very little conclusive evidence. For 
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every study, statistical or theoretical, that contains a proposed 

solution or recommendation, there is always another, equally well 

documented, challenging the assumptions or concl islons of the first. 

No one seems to agree with anyone else's approach* But more dis** » 

tressing I must confess, I stand with my colleagues confused and 

often disheartened* 

Our continuing inability to produce cumulative knowledge and general principles 
has, I believe, led to widespread disillusionment with psychological research 
and consequent reductions in funding for our rerearch by governmental and 
other bodies* 

Psychological researchers have relied on a general two-step procedure to 
produce cumulative knowledge* First, individual scientists conduct numerous 
studies on a given hypothesis — the hypothesis that perceptual speed predicts 
success in clerical work, for example, or the hypothesis that reducing class 
size Increases pupil learning* Second, this body of studies is Integrated 
subjectively and non-quantltatively in a narrative review which is published 
in a review journal such as Psychological Bulletin . This paradigm for the 
production of knowledge has failed* Almost Invariably, the reviewer reports 
that there are inconsistencies, conflicts, and contradictions between findings 
of different studies. The reviewer then concludes that further research is 
needed and that conclusions about general principles are not yet possible. 

Some have hypothesized that the conflicting findings mean that the 
phenomena we study are much more complex than we originally thought, and 
that the complexity of findings is due to the presence of many unsuspected 
interactions (i.e., moderator variables)* But there is another, more 
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parsimonious possibility: the possibility that the information processing 
task imposed in reviewing a body of studies is too complex for the unaided 
human mind. The reviewer must simultaneously execute two very difficult 
tasks. He or she must subjectively integrate findings across numerous 
studies — sometimes hundreds of studies. At the same time, he or she must 
keep in mind and allow for the effects of statistical and methodological 
artifacts which distort study findings — for example, sampling error, measure- 
ment unreliability, and reatriction in range. These tasks are beyond the 
unaided information processing capacities of any human being. 

Failure to produce cumulative knowledge is the most important problem 
today in psychology and the social sciences. The most important recent 
development is therefore the development of quantitative methods which 
solve this problem. These methods lift the information processing burden 
from the reviewer by quantitatively integrating findings across studies 
while simultaneously correcting for the effects of statistical and measure- 
ment artifacts which distort study findings. As Glass (1976) has pointed 
out, these techniques represent the belated application of the same sta- 
tistical methods that we routinely use to analyze data within studies to 
the problem of integrating findings across studies. Application of these 
methods can clarify confusing research literatures and allow the establish- 
ment of general principles. 

Gene Glass at the University of Colorado was the first to systematically 
address this problem. He also originated the term "meta-analysis". He and 
his associates advanced a meta-analytic method composed of the following steps: 

1. All efi'ect sizes are expressed in SD units (or alternatively, 
in correlation form). 
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2, The mean effect size across studies is computed. This represents 
the expected magnitude of a treatment condition or the expected 
size of a correlation. For example, Smith and Glass (1977) fomul 
psychotherapy has an average effect size of .68 standard deviation 
units of the control group. White (1976) found that the mean 
correlation between SES and academic achievement Is .25. 

3, Properties or characteristics on which studies differ are coded 
and then correlated with effect sizes in an effort to find the 
causes of differences between studies in reported effect sizes. 
Studies may differ on age or sex of subjects, methodology used, 
and other variables. Although numero««c variables are typically 
coded, the general finding has been that few are correlated with 
study outcomes. Because the sample size for this type of analysis 
is the number of studies — not the number of people — there are 
often severe problems of capitalization on chance and low statis- 
tical power. For example, there may be 70 studies and 50 study 
characteristics. 

Glass and his associates have applied his methods of meta-analysis to 
a varitey of heretofore confused research literatures. In almost every 
case, the research literature has been clarified and general principles 
have been established. As one example. Glass and Smith (1979) have applied 
meta-analysis to the vast, conflicting, and heretofore uninterpretable lite 
ture on the effects of class size on pupil achievement. Based on 725 
studies, their results revealed a very definite monotonic relation between 
class size and achievement, with the achievement difference ranging up to 
.90 SD units for the smallest (N - 1) vs. the largest (N - AO) classes. 
Further, the effect sizes were larger for the better controlled studies. 



This is the kind of generalization that is needed as a foundation for both 
theory development and social policy decision-making. 

Concurrently with Glass' work and independently of it, John Hunter and 
I and our associates developed our own meta-analysis procedures. Our pro- 
cedures extend meta-analysis by providing methods of correcting for the 
distorting effects of artifacts such as sampling error, measurement unrelia- 
bility, and range restriction while integrating findings across studies 
(Schmidt and Hunter, 1977; Schmidt, Hunter, Pearlman, and Shane, 1979; 
Pearlman et al., 1980). Methodological contributions in this area were also 
made by Callender and Osburn (1980) and Raju and Burke (1983). This pro- 
cedure was originally developed to integrate employment test validities 
across studies, but has since been generalized for application to all research 
areas (Hunter, 1979; Hunter, Schmidt, and Jackson, 1980). Steps in this 
procedure are as follows: 

1. Effect sizes are expressed as correlations or d-values and the 
average effect size is computed across studies. This mean effect 
size is then corrected for the attenuating effects of instrument 
unreliability and range restriction. This is a step not irxluded 
in Glassian meta-analysis. 

2. One then determines whether the variance in effect sizes across 
studies is due solely to statistical and measurement artifacts. 
This step is also not included in Glassian meta-analysis. If one 
can reject the hypotheses that the observed variance of effect 
sizes is greater than the variance expected from artifacts, one 
concludes that the mean corrected effect size estimates the true 
effect size, and a general principle has been established. The 
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mean corrected effect size then Incorporates and summarizes the 
results of all previous studies. 

3. If one cannot reject the hypotheses that the variance of effect 
sizes is greater than the expected from artifacts, one then deter- 
mines whether any of the study characteristics are correlated with 
effect size. This step we borrowed from Glass and his associates 
(while recognizing and warning against the severe problems of capi- 
talization on change and low statistical power). 

4. If the remaining variance Is still too large to be accounted for 
by artifacts, It Is adjusted for the effects of these artifacts, 
and this adjusted variance Is used to set confidence or credibility 
Intervals around the mean effect size. Again, this Is a step not 
Included In Glasslan meta-analysis. 

To date, this procedure has been applied to over 500 research literatures 
in employment selection, each one representing a predictor-Job performance 
combination. These predictors have Included nontest procedures such as eval- 
uations of education and experience and Interviews, as well as ability and 
aptitude tests. In many cases, artifacts accounted for all variance across 
studies; the average amount of variance accounted for by artifacts has been 
approximately 80%. As an example, consider the relation between quantitative 
ability and overall job performance In clerical jobs (Pearlman et al., 1980). 
This substudy was based on 453 correlations computed on a total of 39,584 
people. 77% of the variance In observed validities was traceable to arti- 
facts, leaving a negllble variance of .019. The mean effect size was .47. 
Thus Integration of this massive amount of data leads to the general and 
generallzable principle that the correlation ,*}etween quantitative ability 



and clerical performance Is .47, with very little If any true variation 
around this value. Like our other findings, this finding shows the old 
belief that validities are situ'ationally specific to be false. Cumulative, 
generalizable knowledge is^ possible. 

Current Status of Meta-Analysis 
The response to these new methods by researchers has been very 
gratifying. Here is some evidence of this: 

1. Meta-analytic studies are appearing more and more frequently in 
psychological and educational journals — and not just in review 
journals like Psychological Bulletin . They are also appearing in 
the primary research journals— even though meta-analysis are reviews 
Meta-analysis appears to have increased the status of reviews. 

2. Hardly a week goes by that I do not receive several meta-analysis 
studies from journal editors to review. I'm sure this is true of 
other reviewers also. 

3. There are now four books on meta-analysis: 

A. Glass, McGaw, and Smith (1981) 

B. Hunter, Schmidt, and Jackson (1982) 

C. Cooper (1984) 

D. Rosenthal (1984) 

5. A chapter on meta-analysis has appeared in the Annual Review of 
Psychology (Green & Hall, 1984). 

6. A count revealed that there are at least 26 meta-analysis presenta- 
tions on the program at this convention (not counting this one). 
This count includes only those with meta-analysis in the title and 
Is therefore almost certainly on the low side. 
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8. Our meta-analysis procedures have now been applied to numerous topics 
outside the area of validity generalization in employment selection. 
Some examples include: 

1. Correlates of Role Conflict and role ambiguity ( Fisher, Gettelson, 
198A, and Jackson & Schuler, in press). 

2. Effects of realistic Job previews (Premack and Wanous, 1984; Cascio 
and McEvoy, 1984). 

3. Evaluation of Fielder's theory of leadership (Peters, et al. , 

1984). 

4. Accuracy of self-ratings of ability and skill (Mabe & West, 
1982). 

5. Relation of LSAT scores to performance in law schools (Linn & 
Dunbar, 1981). 

6. Relation of job satisfaction to absenteeism (Terborg, et al. , 

1982) . 

7. Ability of financial analysts to predict stock growth (Coggln 
& Hunter, 1983). 

8. Premorbid functioning and recidivism in Schizophrenia (Stof f elmeyr, 
Dillavou, & Hunter, 1983). 

9. Examples of Application of Glassian meta-analysis methods include: 

1. Effectiveness of computer assisted instructions (Kubik, Cohen, 
& Ebeling, 1979). 

2. Effects of social facilitation on task performance (Bond & Titus, 

1983) . 

3. Sex differences in inf luenceability (Eagly & Carli, 1981). 

4. Effects of psychotherapy (Smith & Glass, 1977; Shapiro & Shapiro, 
1982). 
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5. Sex bias in counseling and psychotherapy (Smith, 1980). 

6. Effects of drug therapy on psychological disorders (Miller, 1977). 

7. Sex differences in ability to use nonverbal cues (Hall, 1978). 

8. Class size and student achievement (Glass & Smith, 1979). 

9. Effects on cooperation and competition on group achievement 
(Johnson, et al., 1981). 

In a talk on the need for meta-analysis that I presented four years ago 
at the 1980 APA convention, I made the following statement: 

"At one time in history of psychology and the social sciences, the 
pressing need was for more empirical studies examining the problem 
in question. But now large numbers of research studies have accumu- 
lated on many research questions. The need today is increasingly 
becoming not additional empirical data but some means of making 
sense of the vast amounts of data that have accumulated. Unless we 
can do this, there is little hope of producing the cumulative gen- 
eralizable knowledge essential for theory development and for the 
solution of social problems. Quantitative techniques like those 
described here provide a solution to this problem. Their applica- 
tion and exploitation is therefore the most pressing research need 
of the 1980's." 

Baaed on developments since then. I believe we are on the way toward 
meeting that need. 
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